Two Stage Max Gain Content Defined Chunking for De- duplication

––Data de-duplication is a very simple concept with very smart technology associated in it. The data blocks are stored only once, de-duplication systems decrease storage consumption by identifying distinct chunks of data with identical content. They then store a single copy of the chunk along with metadata about how to reconstruct the original files from the chunks, this takes up the less storage space than storing the whole thing again. Most of the de-duplication programs are based on fingerprinting. Data de-duplication segments the data into chunks and writes those blocks to disk. And signature is created for each data segment much similar to the fingerprints. And an index is associated for the fingerprint. The index, which can be recreated from the stored data segments. When the repeated new data is encountered in the data stream the de-duplication algorithm inserts a pointer in a metadata which related to the data block which is already present in the data stream. If the same block appears more than once multiple pointers are created for a single block on the disk. This eliminates the redundant blocks in the data stream and improves the storage space efficiency. Content defined chunking means the duplicate blocks are identified based on the internal properties on the data stream. There are many algorithms are in use to identify the repeated content on the data stream. In this paper, a new two stage content based chunking algorithm, the first stage can be any content based algorithm based on the cut points and in the second stage the newly developed max gain algorithm, identifies the new cut points on the repeated data.